Adiabatic layering: A new concept of hierarchical multi-scale optimization

-Recurrent neural networks (RNNs) with linearized dynamics have shown great promise in solving continuous valued optimization problems subject to bound constraints. Building on this progress, a novel method of constrained hierarchical multi-scale optimization is developed that applies to a wide range of optimization problems and signal decomposition tasks. Central to the underlying concept is the definition of adiabatic layering. Analytic justification of this model can be regarded as a natural development of the mean-field theory. What emerges is an alternative hierarchical optimization method that promises to improve upon existing hierarchical schemes in combining the accuracy of global optimization with the compact representation of hierarchical optimization. Whereas conventional hierarchical optimization techniques typically tend to average over fine-scale detail when applied to bound-constrained problems, such behaviour is avoided by the modified dynamics o f the proposed method. Applied to the signal decomposition problem o f RBF approximation, the behaviour o f the adiabatic layering model is shown to be in close correspondence with the theoretical expectations. Keywords---Recurrent neural networks, Hierarchical optimization, Bound-constrained optimization, Multi-scale representation, Mean-field annealing, Adiabatic layering, RBF approximation, Iterative methods. 1. I N T R O D U C T I O N In this paper we study quadratic minimization problems of the form minimize J(x) = ~xTQx -xTy (1) s u b j e c t t o a i < x / < ~ i (iE{1, 2 , . . . ,n}) , (2) where Q E y/n×, is a positive (semi)-definite symmetric matrix, and y E ~n is a constant vector. The system (1) and (2) describes an optimization problem in which it is required to minimize the quadratic function (1) over a convex closed set in a Hilbert space. Equations (1) and (2) are quite general and capture a wide range of numerical problems, defined on geometric domains, for which no closed-form solution is available. Acknowledgements: This research was supported by NFWO/ FNRS, "Multidimensional Image Compression and Restoration for Quantitative Applications", grant NFWO: 2.0041.94, grant NFWO (Nationale Loterij): 9.0051.93, grant FNRS: 9.4594.93. Requests for reprints should be sent to Bart Truyen, ETROIRIS Research Group, Department of Electronics, Vrije Universiteit Brussel, Pleinlaan 2, B-1050 Brussels, Belgium; E-mail: [email protected] Recently, recurrent neural networks with linearized dynamics have successfully been applied to different bound constrained optimization problems (Sudharsanan & Sundareshan, 1991; Bouzerdoum & Pattison, 1993; Truyen et al., 1994). The particular attraction of this network model should primarily be attributed to its linear parameter encoding and the guaranteed feasibility of its solutions. It will be shown in this paper that the linearized dynamics of this network model will lead to a new mechanism of hierarchical multi-scale optimization. The dynamics of the network are expressed as: d = G u + Ss + i b (3)